Model Selection for Part of Speech Tagging With Type Supervision

Model selection (picking, for example, a parametric model family, a prior, and an estimation criterion) is crucial for building high-accuracy classifiers. In supervised learning settings, the accuracy of a model can be estimated on a labeled set and used to guide modeling decisions. In unsupervised or type-supervised learning settings, unsupervised model selection criteria are used, but their performance is far from optimal (Smith and Eisner, 2005). Here we propose a new model selection criterion for type-supervised sequence labeling settings, which uses the available weak supervision (type-level constraints) more directly to come up with an accuracylike metric. We evaluate the effectiveness of the method on type-supervised POS-tagging in nine languages, using both HMM and LDA-based models, and show that it outperforms two unsupervised model selection criteria.